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ABSTRACT 

This paper presents results of a study examining the 
relationship between the American Board of Surgery's In-Training 
Examination and its Qualifying Examination. 19^82 Test scores on both 
examinations of 764 candidates .in their fifth year of training as 
Chief Residents were analyzed. Descriptive statistics, correlations, 
and standard errors were computed. A moderately high correlation 
coefficient was observed. The findings do not support t^ose 
-prevxously reported in the literature. Regression equations and 
theoretical expectancy tables were prepared, for use by residency 
program directors in predicting future fucqess in the certification 
process. Various factors that possibly influenced the size of the 
correlation coefficient are discussed, inc^luding examination 
purpose/content, security breaches, motivations. to take the 
examination, and other factors. Implications for other medical 
specialties in analyzing and interpreting I|»-Training Examination 
results are discussed. (Author/BS) \ 
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ABSTRACT 

n 

A Study of the Relationship Between a Medical Certification Examination and an 
In-Training Examination. 

■ $ 

Thomas W. Blester, American Board of Surgery 

\. : / • ' o 

■ • • • 4 

This .paper presents results of a study examining the relationship between the 

4 • 

American Board of Surgery's In-Training Examination arid Qualifying 
Examination. A moderately high correlation coefficient was. observed. The 
findings do not support those previously reported in the literature. 
Regression equations and theoretical %xpectauty tables were prepared for use 
by residency program directors in predicting future success in the 
certification process. Various factors that possibly influenced the size of 
I the correlation coefficient are discussed, inclxiding examination 
I purpose/content, security breaches, motivations to take the examination, and 
[ other factors. Implications for other medical specialties in analyzing and 
interpreting In-Training Examination results are discussed. 



\ 



INTRODUCTION 



Similar to guidelines for other medical specialties, the "Essentials for 
Approved Residencies" in Surgery require that program^ provide a blear 
demonstration of each resident's competence in basic surgical knowledge befc 
progressing to the next level of sui)ervised semi-independent operative 
experience and patient management. When these guidelines were first published 
in 1974, program directors in surgery urged the American Board of Surgery 
(ABS) to develop a written examination, for the assessment of surgical 
residents. The request came at a time when the Board was becoming concerned 
about the relatively high failure rates .on its certification examinations and 
about what it perceived as deficits in the cognitive knowledge base of many 
residents. The first In-Training Examination (ITE) developed by the Board was 
administered in 1975. This paper examines the use of the ITE for resident 
e\aluation after ten years in the field and, in particular, the relationship 
between ITE .results and performance on the Board's Qualifying Examination 
(QE), the written examination (Part I) necessary for certification in Surgery. 



BACKGROUND ON IN-TRAINING EXAMINATIONS . 



In-Training Examinations sponsored by medical specialty organizations date 
.back to the early sixties in orthopaedic surgery and neurological surgery 
(Levit, 1969). The specialty boards for these two disciplines were concerned 
about the high failure rate of candidates appearing for certification, despite 
the uniformity of the prescribed period of training. .The 1964 In-Training 
Exai^iination for these specialties was developed with the assistance of the 
Natjjonal Board of Medical ^jcaminers. 



In-Training Examinations are generally oriented toward three primary goals: 1) 
'TO define a content domain of knowledge necessary for competent practice; 2) 
to assess the relative strengths and weaknesses of individual residents at a 
time early enough in their training that deficits can be corrected; and, 3) to 
help directors of residency programs detect peas of relative strengths and 
weak®sses of their program (Grosse, Cruft, and Blaisdell, 1980). It is 
Important to keep these goals in mind. The first ABS ITE was welcomed by 
program directors as a valuable formative tool (Friedmann, 1985). r 

The American Board of Medical Specialties reports that written In-Training 
Examinations are now offered in 17 of the 23 -recognized medical specialties, 
with three others currently in the planning stage -(ABMS, 1984). Table 1 lists^ 
the specialties that now administer ITEs. Seven of these examinations are 
sponsored by the specialty board, with the others sponsored by a professional 
society, academy, or college. All are administered annually,' and all are open 
to residents, with many also open to practitioners. Many, such as the ^ABS 
ITE, are designed primarily for use by the residency program director, rather 
than the residents themselves. Internal Medicine is a noteworthy exception to 
the specialties offering written ITEs, instead requiring a detailed resident 
evaluation system as a \^ey element of thie credentialing process. Iii some 
cases (e.g.. Surgery), I^ps are developed through the traditional committee 
approach while in others (e.g. , , Pediatrics) , examinations are comprised of 
item§ used in other examinations, such as prior certifying examinations. 
Specialty In-Training Examinations are distinguished from the . self-assessment 
programs/examinations offered by the professional societies tn most 
specialties. While the overall goal is diagnosis and self-improvfement , the 



TABLE 1 



MEDICAL SPECIALTIES OFFERING iN-TRAINING EXAMINATIONS . 
AND/OR SELF-ASSESSMENT PROGRAMS /EXAMINATIONS 



Speci alty 

Allergy & Immunology 

Anesthesiology 

Colon & Rectal Surgery 
Dermatology 
Emergency Medicine 
Family Practice . 
Internal Medicine 
Neurological Surgery 
Nuclear Medicine 
Obstetrics & Gynecology 
Opthalmology 
Orthopaedic Surgery 
Otolaryngology 
Pathology 
Pediatrics 

Physical Medicine & Rehabilitation 
Plastic -Stirgery 
^iPreKfentive Medicine 
Psychiatry & Neurology 
Radiology 
Surgery 

Thoracic Surgery 
Urology 



In-Training Self-Assessment 
Examination Sponsor* Program/Examination 



yes 
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yes 


yes 
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yes 
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yes 
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yes 


yes 
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yes , , 
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yes** 


yes 
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yes 
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yes 


yes 
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yes 


yes 


B 




yes 


S 


yes- 



* Sponsor Key: B - Board 

S = Professional Society, "^College, Academy 

** Not open to residents 

Source: American Board of Medical Specialties 1984 Annual Report 




'A 



latter generally lack standard adi^inlstration conditions and national score 
norms. They are offered to indivliduals rather than residency program 



directors and are usually targeted at 
of a junior resident. 



a knowledge level somewhat beyond that 



IN-TRAINING EXAMI^lt?S | AND THE CERTIFICATION PROCESS 

~ ^ — V\f\ : • 

the three primary goals listed abov^- illustrate the basic formative nature of 
In-Training Examinations, namely diagnosis of weaknesses for self-improvement 
prior to completion of graduate medical training. Except for two specialties 
(Anesthesiology and Neurological Surgery), the ITEs are not an official part 
of the credentialing process. However, in no case is an ITE officially used 
to evaluate a program and results are not reported to Residency Review 

c 

Committees (RRCs) , unless the program director voluntarily elects to do so. 

Still, since RRCs for most medical specialties use the number of residents who 

■"\ . ■ ■ 

'»)■•■ . 

successfully complete certification requirements for the particular specialty 
board as one important cMterion for program accreditation, residency 
directors, responsible for assuring the professional Competence of their 
graduates, have often expressed an interest in knoi^ng the relationship 
between In-Training Examinations and board Certifying Examinations. If the 
relationship is strong and both examinations are content valid, the ITE is a 
useful tool, for program directors in determining ^ that their residents have 
acquired the basic knowledge needed for competent practice tfi the field • 
Careful monitoring of progress, through ITEs and other releyant assessment 
procedures throughout the course of residency, helps to ensure that graduates 
aire knowledgeable^ in their field and certification success rates, as 



ERLC 



validation criteria, will be^^ high. More important, graduates will be 

. • • i ' ■ "' ■ 

competent practitioners. 

• ' ■ • ■ '^-^ .V. ■ • 

Shetler (1982) investigated the relationship between the ABS ITE and the ABS 
QE, the initial step in the cjartif Ication process. This study looked > at 
results from- one particular residency program. He reported a cprrelation of 
.79 and suggested guidelines for program directors on how to use and interpret 
ITE results. However, the usefulness of this study's findings is somewhat 
limited because of the small sample size and some other technical flaws (e.g., 
use of percentile scores as a basis for calculations) . 

Garvin aiii Kaminsky (1984) followed Shetler 's model wit^ from their 

program over a five-year period. Their findings were virtually identical to 
Shetler's, with a correlation of .76. Results of both; of these /Studies 
received widespread circulation and are frequently cited by program directors 
in surgery. However, since some questioned the validity of findings and 
because of the small sample size, the ABS decided to conduct further 

■ ■ ■ J ' 

investigations of the relationship between its In—Training and Qualifying 
Examinations'. 



STUDY OBJ'ECTIVES 



The current study was designed to enhance the findings of the Shetler and 

Garvin-Kaminsky studies by examining ITE and QE results for all surgery' 

residents participating in the ABS. assessment program. Specif ically,/ thre€f 

majbr objectives or research questions were addressed: / 

1) What is the observed relationship between the America^ Board 
Surgery's In-Training Examination and Qualifying Examination? 



2) What factors seem to influence the strength of the relationship 
between the ITE and the QE? 

• ' • ■ ' " 

3) What are the implications of the relationship for residency program 
directors in surgery? 

Although the objectives are essentially the same as those addressed by the 

reported studies, findings . are more generalized and accurate. The findings 

and methodologies are also somewhat generalizable to other medical specialty 

areas that administer ITEs, suggesting a fourth objective! 

4) What are the implications of the study for other medical specialty 
\ areas? - ^ 

■ METHODS AND DATA SOURCES • 

. In 1982, the ABS ITE was administered to 7349 resident^, representing 
virtually all of the accredited surgery residency programs nationally. Of 
this number, 915 were in their fifth year of training as Chief -Residents . A 
total of 764 of these Level V candidates .also took the ABS QE in 1982. Test 
scores for this group on both examinations were analyzed. Standard scores, 
based on a me^ of 500 and a standard deviation of 100, were used as a basis 
for the analys'is. Descriptive statistics, correlations, and standard errt^rs 
were computed. A scatter plot w^s prepared anJ| a regression equation for 
predicting an individual's QE score on the basis of the ITE score was 

: constructed. Finally, a theoretical expectancy chart showing the probability 
of passing the QE at different level score i,ntervals on the ITE was prepared. 

RESULTS 

Prior to formal data analysis, the psychometric properties of both 
examinations were considered. The 1982 ITE had 351 scorable units (although 
200 items, many had several scorable units), with an average difficulty or "p" ' 



value of .72 and overall reliability 'of .97. The 1982 QE had 770 scorable 
^nits (302 items), with an average "p" of .74 and overall reliability of .99; 
The high reliabilities indicate that the precision of both examinations . is 
excieptionally good and one should be fairly confident about individuals' 
scores. \ 

" ' ■ \ ' 

The^ means for the study group were 589 and 557 for the ITE and QE, 
respectively. The high standard score for the ITE was as expected, since 
Level^V scores are much higher than the total [group taking the ITE. The high 
mean score for the QE indicates_tha^ the. study ' group is not representative of 
the total group taking the examination, but is fairly representative of the 
group of U.S. /Canadian medical schodl graduates taking the test for the first, 
time. The score ranges were 370-732 for the ITE and 225-715 for the QE. The 
ITE range was somewhat restricted, as expected, because of the nature of the 
resident group. ^ 

The observed correlation coefficient between the two examinations was .48. 
This is contrasted to coefficients of .79 and .76 found in earlier studies. 
Thus, while these authors concluded that the correlation between the ABS ITE 

o 

^ ■ . \ 

and QE was very hig^, the conclusion drawn from the Board study is that it is 

only moderately high (onlji 23% of the variance is explained). The regression 

equation for predicting the QE score on the basis of the ITE score is' as 

follows: 

/Si * 

QE = .575 ITE (V) + 218.15. 

However, the standard error of estimate is fairly high ^t about 50 points. 

Thus, the predicted score range would be at least ipO points, probably not 

that useful for program directors. \ ^ 

10 



Most studies' of this type reported . in the literature use the regression 
approach, for predictijon . However, as Guion (1965) noted in his Personnel 
Testing textbook, such an approach is not most appropriate for making 
categorical predictions. Such is the case when trying to predict passing or 
failing the Qualifying or other certification examination on the basis of 
In-rraining Examination results. For this purpose, Guion recommends 
theoretical es^pectancy charts. Table 2 shows the QE pass/fail expectancies 
for the ABS given ITE scores at >?|arious intervals. This chart would lead one 
to choose a score of about 490-510 for a Level resident as the basis for 
predicting the QE. - Even with a\ moderate correlation, one could correctly 
predict passing or failing the QE \at least 85^ of the time using this chart. 
Such a chart would be very useful for directors of surgery residency programs, 
in addition, such sin approach, altho\igh fairly simple, would jbe useful for all 
residency programs. Directors shoulipi also consider typical score trends in 
assessing growth through each levei^l ^f the residency program. If scores at 
lo^^r levels are beloTw^ that for which a passing score at tlie Senior Resident 
level would be predicted, appropriate remedial education could be prescribed. 

' \ ■ - , ■ ■ . . 

f , DISCUSSION 

The correlation results are disappointingly low, particularly in light pf the 
high coefficients observed in the earlier sttidies. One major reason for the 
fairly low. observed correlation is the probability of security breaches in the 
administration of the ITE. For example, it \ has been reported that some 
program directors may allow open-book administration or proctors may not 
strictly adhpre to test administration guidelinesu ^ addition, for soi&e time 
it has been clear that copies pf prior ITEs have bteen reproduced and 

11 



TABLE 2 



EXPECTANCY CHART FOR PREDICTING PASS/FAIL ON Afes QUALIFYING EXAM 

1 - " ' 

ON BASIS OF ABS ITE (LEVEL V) SCORE* 



ITE Score 
Range 

(Level V) ^ 



# of .. 
Examinees 
(1982)** 



Proportion 
Passing QE 
(@495)*** 



Proportion 
Falling QE 
(@495) 



711 + 




5 . - 


M.OO 


0.00 


671 - 


710 , 


42 


';^3 


.07 


631 - 


670 \ 


142 


.92 


.08 


591 - 


630 


\ -218 


.90 


.10 


551 - 


590 ' > 


\ 18L 


.86 




511 - 


550 


. 110 


.85 


.15 


471 - 


510 


39 


.46 


.54 


431 - 


470 


19 


.37 


. .63 




430 




.25 


.75 


, TOTAL 




764 . 


.84 


.16 



Observed correlation between ITE (Level V) and QE = .48 . 

Regression equation for predicting QE score from Level C ITE score: 

QE = .575 ITE (V) + 218.15 . 

Standard error of estimat^i^ = 50*45 ; thus, predicted QE shoui^ be in the range 
of QE ± 50.45 roost of the time. ' . \ 



* Based on empirical results from 198^. 

** Study group includes all examinees who took both ITE and QE in 1982. 

*** Passing score of 495 in 1982 represents 1.08 standard deviation below the 
mean of the reference group. % 



« 
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distributed throughout jnany parts of the country. Since previously used 
questloas comprised about 57% of the ITE content in 1982, results might not 
ac9urately represent the true knowledge of all Level V candidates. Indeed, a 
recent ABS ^tudy of performance on new versus used items on the 1984 ITE 
indicated "suspicious" scores for about 20% of the surgical residency 
programs. This would introduce a large error factor into the score results 
and the correlation would be significantly reduced. Analyses showed that ITE ( 
results for residents in these ^programs could be inflated by 50 to 125 
. standard score points. Security measures were taken ^for the 1985 examination 
and ^a follow-up study examining results for selected programs with no apparent 
security problems is currently underway i^^-order to gain a more valid ^itnfure ^ 
of the relationship between the ITE am^ the QE. 

. Residency programs for other specialties should likewise be aware of tljie 
problems that breaches of examination security pose for accurately 
interpreting ITE results. Friedmann (1985) indicated that seven specialties, 
including surgery, reported security problems associated with their ITEs. 
Although many program directors become incensed at charges of cheating on the 
part of residents, Ballinger (1985) and Friedmann, provide good historical 
overviews that help explain why such incidents may be increasing. Although 
their focus is on the surgery examinations, their comments apply as well to 
other specialties. In earlier years^Cs^. , early to |mid seventies), not much 
importance.. was attached to the ITEs; they w^r^used primarily in a formative 
sense to diagnose weaknesses in the residents' knowiedg€L^bas^e - Times changed, 
however, and the examinations, in some instances, hf|ve /become pl^To^al^i^ t^ 
lives of residents, particularly in pyramidal programs, who are competing for 
^ fewer positions, in fewer programs, where career success may hinge on test 
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outcoinesL In addition, as the nuiAber of programs dwindled-, program directors 
becam^^jjfoncerned that they would be judged (e.g., by the RRC) on how their 
residents performed on the ITE. Clearly, the attitudes and motivations of 
program directors and residents strongly influence performance on the 
examination. * y"' • 



Berry (1984) noted/ y;drious uses laxid abuses of the ABS ITE, including the 

increasing . shift jfiom formative to summative interpretations. This shift has 

- ■ 

led to a great deal of confusioh and anxiety, A Board survey of program 
directors indicated that 34% used a formal cutting . score for determining 
passing or failing the ITE. Most (85%) noted that they consider test results 
in residen^^ promqti^g^ decisions. Five other Boards reported setting formal 
passing scdi^s- for thelr^:,£f|jsv^^ Although the surgery program directors 

indicated that they use the ITE for formative evaluation of their resiHents, 
the three primary goals apparently need to be reeidphasized. 

While ABS ITE results are reported in -a norm-referenced sense (e.g., standard 
scores and national, percentiles) , results By program and individual resident 
are also reported in a criterion-referenced sense. Kejrwords that describe the 
specific items answered incorrectly by e^ch .e3camine<i and the entire group of 
examinees are reported to program direc4ors. An exaniple is included in the 
Appendix. TJxese results can be 'used/ for overall program improvement and 
individual remediation. In this regard. Dean, Hanni, Pyle, and Nicholas 
.(19M) rej^rted positive effects of Coaching and remedial programmed 
instruction on ABS ITE scores. Program directors and residents alike have 
repeatedly requested that copies of previous examinations be made available 
for self study. However, this would create tfel need for. new test forms each 



year, would probably cause some equating and norming problems, and would be 
very expensive. In addition, the SESAP ekaminaitons of the American College 
of Surgeons serve this purpose* 

Another possible cause for the low correlation, somewhat related to the above 
discussion, is the varying amount of preparation that residents make for the 
examination. Since residents differ with regard to their motivation to 
succeed on the examination, some prepare extensively for it while others do 
not. An ABS survey of chie'^f residents indicated that 51% actively studied for 
the ITE prior to administration. While Donovan (1985) notes that "cramming" 
should not significantly affect_ performance, the results of the study by Dean 
and associates may' suggest otherwise . Some program directors have indicated 
that formal study groups have successful payoffs while others prefer not to 
encourage special cramming but rather to assess the level of knowledge 
retained throughout the normal course of the residency (in other words, use * 
the ITE as a spot check) . 

Both the ABS ITE arid QE are designed to assess examinees' level of knowledge 
related to surgery, in order to assure that all residents completing training 
possess the essential or "core" knowledge necessary for competent practice. 
As such, one might expect correlations to be high. Although each examination 
'is constructed by a different committee, using a slightly different^^ontent 
outline, the basic content categories are fairly similar. However, a^ Donovan 
(1985) notes, the distribution of questions on the ABS ITE is very different 
from the QE. The former is aimed at residents completing their second year of 
graduate training while the latter is appropriate for someone who has just 
finished the fifth year of residency. A large part of the ITE is oriented 
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toward factual recall of basic science conc^epts rather than the evaluation of 
judgment and patient management (not PMPs, however) emphasized in the QE. A 
frequent criticism of the ITE from program directors and residents is that it 
should contain more patient management items. There are some other content 
differences, as well. For example, the ITE has a heavy emphasis on knowledge 
of "the body as a whole" while the QE deals more with diagnosis and treatment 
of more specific organ systems. The audience for each particular examination 
~is-~impo«:ant to keep in mind, since surgery residents gradually acquire more 



intensive clinical experience in their fourth and fifth years. Clarke (1982) 
documented increasing decision-making skills from junior to chief-residency 
levels in surgery. 



Studies by the American Board of Internal Medicine found that multiple-choice 
questions assessing knowledge, synthesis, and clinical judgment measure 
essentially the same aspects of clinical competence (Norcini, Swanson, 
Webster, and Grosse, 1984). However, these studies were conducted within the 
same examinee groups, unlike the case described above for surgery where the 
target audiences and corresponding hierarchial level of cognitive achievement 
differ. . / 

Other studies that have examined the relationship between kno^^dge as 
measured by multiple-cl?oice examinations and other measures of /clinical 



competence may shed some light on the observed ITE-QE correlations/. Lazar, 
DeLand, and Tompkins (1980) studied the relationship between ABS ITE scores 
and other measures of clinical competence, finding a very low /correlation 
(.27). ^ They concluded that this low correlation was desirable since the 
indicators were measuring different aspects of resident competence. Similarly 
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,low correlations between multiple-choice test scores and clinical competence 

ratings were reported by Stillman (1984) and Berenson, Stiinmel, and ^fses 

• ■■ ■ . ■ . ■ •/ ■ 

(1981). In the Board survey of program directors, 58% reported that the ITE 
results correlate* well with their impressions of their residents' clinical 
performance. ^ 

From these studies and the observed ITE-QE correlation one might conclude that 

the examinations are, in fact, measuring different, buf somewhat related 

attributes. Clarke (1982) stated that tjie satisfactory surgery resident 

requires both factual knowledge and judgment as well as the ability to combine 

these assets into clinical decision making. The ITE, while providing a valid 

assessment of residents' requisite factual knowledge (as indicated, by 96% of 

surveyed program directors), probably does not adequately tap judgment and 

clinical decision-making. As such, Lazar and associates emphasized that the 

ITE cannot be used as a sole basis for assessing residents' clinical 

competence. Shetler and Berry echo this sentiment, and, likewise, the Board 

has taken a strong position encouraging program directors to use multiple 

'v * 
criteria in assessing the competence of their residents. 

Finally, various statistical factors may contribute to the relatively low 
correlation. These include restriction of range and probable ceiling effects. 



SUMMARY AND CONCLUSIONS 




Previously reported findings concerning the degree of the relationship b.etween 
the ABS JTE and QE were not observed. The actual correlation was only 
moderately high. Apparently, the two examinations are measuring somewhat 

■ 17 



different attributes. Other factors that possibly account for the lower 



correlation are security problems, motivation to take the examinations, uses 
and abuses of results by program directors, the nature of residency training, 
and statistical factors. Since the ph:|j^osophy and general content guidelines 
of ABS* ITEs are similar to the In-Training examinations of other specialty 
boards, results are generalizable to the other medical specialties, as well. 
Factors\ffecting the scores on the ABS ITE probably also affe^ct ITE scores 
for other medical specialties. In addition, the .4tudy provides these 
special^es with a useful methodology for exploring the relationship between 
their in-training assessments arid certification examinations. In particular, 
the following^ implications can be drawn from the study: / 

• Program directors need to keep the basic goals of In-Training 
Examinations in mind. In the formative sense, results can be used to 
direct remedial study for residents and programmatic deficiencies can be 
pinpointed and corrected. Further development of criterion-referenced 
reporting approaches is encouraged. 

• While the assessment of residents' cognitive knowledge is important, 
program directors need to keep in mind that other measures of competence 
are likewise important. These ather attributes are probably not assessed 
in typical written In-Training Examinations. New tools for the 
assessment of resident competence need to be developed where current 
state-of-the-art is inadequate. 

• Program directors should be sensitive to the possible abuses of 
In-Training Examination results. They should also recognize that results 
are primarily for their use and Residency Review Committees do not 
require actual scores as part of the accreditation process. 

• Program directors should be sensitive to security issues involved in the 
administration of the In-Training Examinations. To the extent that 
security problems exist, scores and national percentiles may be somewhat 
inaccurate. 

• The regression approach for predicting outcomes such as specialty 
certification may not be the most useful or appropriate one for program 
directors . Rather, an approach using theoretical expectancy charts is 
recommended. 



Further studies of trends in the growth of residents' knowledge 
throughout the course of their training need to be conducted. 



In-Training Examination results provide a useful tool for monitoring the 
quality of graduate medicall education and assuring that residents acquire the 
essential knowledge needed to practice their chosen medical specialty. As a 
tool, the examination iriust be used properly. Test scores provide an Important 
piece of Information about the resident, but that information "needs to be 
considered along with nany other data sources concerning the residents' 
knowledge, skills, attitudes, and other behavior*. As Shetler .noted, 
In-Training Examinations will "never completely replace the disc^ning 
intuition of a sagacious teaching staff committed to their residents, their 
program, and the medical discipline. 



/ 

/ 
/ 

•/ 
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